Electronic apparatus and method

ABSTRACT

According to one embodiment, an electronic apparatus includes a hardware processor. The hardware processor is configured to display a first three-dimensional object and a second three-dimensional object on a display, spectroscopically display a first user interface if the first three-dimensional object is selected, rotate the first three-dimensional object and the first user interface through a first angle corresponding to changes in positions of hands of the user, spectroscopically display a second user interface if the first and second three-dimensional objects are selected, and rotate the first and second three-dimensional objects and the second user interface through a second angle corresponding to changes in the positions of hands of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/261,219, filed Nov. 30, 2015, the entire contents of which areincorporated herein by reference.

FIELD

Embodiments described herein relate generally to an electronic apparatusand a method.

BACKGROUND

Electronic apparatuses, such as notebook and desktop computers (PC) anddigital televisions, are well known.

In recent years, such electronic apparatuses have come to employ adisplay panel (3D display) corresponding to 3D display, whereby caseswhere a three-dimensional object (hereinafter, referred to as a 3Dobject) is stereoscopically displayed have increased in the fields ofhealthcare, computer-aided design (CAD), video editing, etc.

In these fields, there are many cases where it is necessary to rotate a3D object stereoscopically displayed on a 3D display, in order to see(observe) the object in detail from various angles.

Therefore, there is a demand for a technique of enabling an operation(via a user interface) of rotating such a 3D object to be intuitive andeasy.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 is a perspective view showing an appearance example of anelectronic apparatus according to an embodiment.

FIG. 2 is a block diagram showing a system configuration example of theelectronic apparatus.

FIG. 3 is a block diagram view showing a functional configurationexample of the electronic apparatus.

FIG. 4 is a flowchart showing an example of a procedure of processing bythe electronic apparatus.

FIG. 5 is a view for explaining a macroframe UI and a microframe UI.

FIG. 6 is a view showing display examples of microframe UIs that includerespective 3D objects.

FIG. 7 is a view for explaining a microframe UI in a situation where two3D objects overlap.

FIG. 8 is a view for explaining an example of a rotational operation.

FIG. 9 is a view for explaining an example of an enlargement/reductionoperation.

FIG. 10 is a view for explaining an example of a moving operation.

FIG. 11 is a view for explaining an example of a user operationperformed to change the center of rotation.

FIG. 12 is a view for explaining an example of an operation of adjustingthe depth-wise position of the center of rotation.

FIG. 13 is a view for explaining an example of a result of the operationof adjusting the depth-wise position of the center of rotation.

FIG. 14 is a block diagram showing a system configuration example of anelectronic apparatus provided with a depth sensor.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings.

In general, according to one embodiment, an electronic apparatusincludes a display configured to display a three-dimensional object, acamera configured to capture an image including hands of a user, and ahardware processor. The hardware processor is configured to display afirst three-dimensional object and a second three-dimensional object onthe display, spectroscopically display a first user interface of arectangular shape surrounding the first three-dimensional object, if thefirst three-dimensional object is selected, rotate the firstthree-dimensional object and the first user interface through a firstangle corresponding to changes in positions of hands of the user thatoccur when the first user interface is displayed, spectroscopicallydisplay a second user interface of a rectangular shape surrounding thefirst and second three-dimensional objects, if the first and secondthree-dimensional objects are selected, and rotate the first and secondthree-dimensional objects and the second user interface through a secondangle corresponding to changes in the positions of hands of the userthat occur when the second user interface is displayed.

FIG. 1 is a perspective view showing the appearance of an electronicapparatus according to the embodiment. This electronic apparatus may berealized as an arbitrary electronic apparatus equipped with a display,such as a notebook computer used by a user. In FIG. 1, it is assumedthat the electronic apparatus is realized as, for example, a notebookcomputer. In the description below, the electronic apparatus of theembodiment is assumed to be realized as the notebook computer.

As shown in FIG. 1, the electronic apparatus 10 includes a main body 11(computer main body), and a display unit 12.

The main body 11 has a thin-box-shaped housing. The display unit 12 isattached to the main body 11 so that it is rotatable between anoperation position in which the upper surface of the main body 11 isexposed, and a closed position in which the upper surface of the mainbody 11 is covered with the display unit 12.

The display unit 12 incorporates a display (hereinafter, referred to asa 3D display) 12A capable of displaying, for example, a plurality ofthree-dimensional objects (hereinafter, referred to as 3D objects). The3D display 12A includes a liquid crystal display (LCD) and a lens unit(not shown). The lens unit is pasted on the LCD. The lens unit includesa plurality of lens mechanisms for projecting, in respectivepredetermined directions, a plurality of structures corresponding to aplurality of pixels contained in an image (3D object) displayed on theLCD. By displaying an image for the left eye and an image for the righteye on the 3D display 12A, the user can perceive a 3D object. The 3Ddisplay 12A may be configured to be able to switch display between a 3Dobject and a 2D (two-dimensional) object.

Assume that the 3D display 12A adopts, for example, a naked-eyestereoscopic vision scheme (for example, an integral imaging scheme, alenticular scheme, a parallax barrier scheme, etc.). In the naked-eyestereoscopic vision scheme, an image for the left eye and an image forthe right eye, which correspond to a 3D object, are displayed in certainpositions on the 3D display 12A. The naked-eye stereoscopic visionscheme enables the user to perceive a 3D object with their naked eyes.

For display of a 3D object, a shutter scheme (time division system), forexample, may be adopted. In the display of a 3D object in the shuttersystem, stereo pair images including image data for the left eye andimage data for the right eye are used. For instance, the LCD is drivenat a refresh rate (for example, 120 Hz) twice the normal refresh rate(for example, 60 Hz). Frames for the left eye included in the image datafor the left eye, and frames for the right eye included in the imagedata for the right eye are alternately displayed on the LCD at, forexample, the 120-Hz refresh rate. 3D glasses (not shown), such as liquidcrystal shutter glasses, enable the user to see, with the left eye,images corresponding to the frames for the left eye, and to see, withthe right eye, images corresponding to the frames for the right eye. The3D glasses may be configured to receive, from the electronic apparatus10, synchronization signals indicating the display times of the left-eyeframes and the right-eye frames, using, for example, infrared rays. Ashutter for the left eye and a shutter for the right eye in the 3Dglasses are opened and closed in synchronism with the respective displaytimes of the left-eye frames and the right-eye frames.

Furthermore, a polarization system, such as an Xpol (registeredtrademark) system, may be used for the display of a 3D object. In thiscase, a group of interleave frames obtained by, for example,interleaving images for the left eye and images for the right eyescanning line by scanning line is generated and displayed on the LCD. Apolarization filter covering the screen of the LCD polarizes, indifferent directions, images for the left eye displayed at, for example,lines of odd numbers on the screen of the LCD and images for the righteye displayed thereon at lines of even numbers. The polarized glassesenable the user to see the images for the left eye with the left eye,and the images for the right eye with the right eye.

A camera 12B is provided on the upper portion of the display unit 12.More specifically, the camera 12B is located on, for example, the frontsurface of the electronic apparatus 10, where the user of the electronicapparatus 10 can be imaged. The camera 12B has a function ofthree-dimensionally imaging, for example, the motion of both the handsand fingers of the user.

A keyboard 13, a touchpad 14, a power switch 15 for turning on and offthe electronic apparatus 10, loudspeakers 16A and 16B, etc., arearranged on the upper surface of the main body 11.

Further, the electronic apparatus 10 is configured to receive electricpower from a battery 17. In the example of FIG. 1, the battery 17 isbuilt in the electronic apparatus 10.

A power supply connector (DC power supply input socket) 18 is providedin the main body 11. More specifically, the power supply connector 18 isprovided at a side, for example, the left-hand side, of the main body11. An external power supply unit is detachably connected to the powersupply connector 18. An AC adapter can be used as the external powersupply unit. The AC adapter is a power supply unit that convertscommercial electric power (AC power) into DC power.

The electronic apparatus 10 is powered by electric power supplied fromthe battery 17 or from an external power supply unit. If no externalpower supply unit is connected to the power supply connector 18 of theelectronic apparatus 10, the electronic apparatus 10 is powered by theelectric power supplied from the battery 17. In contrast, if theexternal power supply unit is connected to the power supply connector 18of the electronic apparatus 10, the electronic apparatus 10 is poweredby the electric power supplied from the external power supply unit. Theelectric power supplied from the external power supply unit is also usedto charge the battery 17.

A plurality of USB ports 19, a High-definition Multimedia Interface(HDMI) (registered trademark) output socket 20, and an RGB port 21, forexample, are further provided in the main body 11.

FIG. 2 shows the system configuration of the electronic apparatus 10 ofFIG. 1. As shown, the electronic apparatus 10 includes a CPU 111, asystem controller 112, a main memory 113, a graphics processing unit(GPU) 114, a sound controller 115, a BIOS-ROM 116, a hard disk drive(HDD) 117, a wireless LAN module 118, a USB controller 119, anembedded-controller/keyboard-controller IC (EC/KBC) 120, a power supplycontroller (PSC) 121, a power supply circuit 122, etc.

The CPU 111 is a hardware processor configured to control the operationof each component in the electronic apparatus 10. The CPU 111 executesvarious computer programs loaded from a storage apparatus, such as theHDD 117, to the main memory 113. The computer programs include anoperating system (OS), a program (hereinafter referred to as a UIprogram) for enabling the user to operate a 3D object, other applicationprograms, etc.

The CPU 111 also executes a basic input/output system (BIOS) stored inthe BIOS-ROM 116 as a nonvolatile memory. The BIOS is a system programfor hardware control.

The system controller 112 is a bridge device configured to connect theCPU 111 to each component. The system controller 112 contains a SerialATA controller for controlling the HDD 117. Furthermore, the systemcontroller 112 communicates with each device on a Low Pin Count (LPC)bus.

The GPU 114 is a display controller configured to control the 3D display12A (LCD) used as the display monitor of the electronic apparatus 10.The GPU 114 generates a display signal (LVDS signal) to be supplied tothe LCD 12A, based on display data stored in, for example, a videomemory (VRAM) 114A.

The GPU 114 can also generate an HDMI video signal and an analog RGBsignal from the display data. The HDMI output socket 20 can send an HDMIvideo signal (non-compressed digital video signal) and a digital audiosignal to an external display through a single cable. The analog RGBsignal is supplied to the external display via the RGB port 21.

An HDMI control circuit 130 shown in FIG. 2 is an interface configuredto send the HDMI video signal and the digital audio signal to theexternal display through the HDMI output socket 20.

The sound controller 115 is a sound source device, and outputs audiodata as a reproduction target to, for example, loudspeakers 16A and 16B.

The wireless LAN module 118 is a module configured to execute wirelesscommunication of, for example, the IEEE 802.11 standard.

The USB controller 119 communicates with an external device connectedthrough the USB port 19.

The EC/KBC 120 is connected to the LPC bus. The EC/KBC 120 isinterconnected to the PSC 121 and the battery 17 through a serial bus,such as an I²C bus.

The EC/KBC 120 is a power management controller configured to executepower management of the electronic apparatus 10. For example, it isrealized as a single-chip microcomputer containing a keyboard controllerfor controlling the keyboard (KB) 13, the touchpad 14, etc. The EC/KBC120 has a function of turning on and off the electronic apparatus 10 inaccordance with a user operation of the power switch 15. On/off controlof the electronic apparatus 10 is realized by the cooperation of theEC/KBC 120 and the PSC 121. Upon receiving an ON signal from the EC/KBC120, the PSC 121 controls a power supply circuit 122 to turn on theelectronic apparatus 10. Similarly, upon receiving an OFF signal fromthe EC/KBC 120, the PSC 121 controls the power supply circuit 122 toturn off the electronic apparatus 10.

The power supply circuit 122 generates electricity (operation power Vcc)to be supplied to each component, using electricity supplied from thebattery 17 or from an AC adapter 140 connected as an external powersupply unit to the main body 11.

The camera 12B shown in FIG. 1 is connected to the system controller112. An image of an object imaged by the camera 12B is used to accept auser operation on the above-mentioned 3D object.

FIG. 3 shows the functional configuration of the electronic apparatus 10according to the embodiment. As shown in FIG. 3, the electronicapparatus 10 includes an image acquisition module 201, an operationdetector 202, a UI generator 203, and a deformation processor 204.

In the embodiment, assume that part or all of components 201 to 204 arerealized by the CPU 111 executing the above-described UI program(software). Further, part or all of components 201 to 204 may berealized by hardware, such as an integrated circuit (IC), or may berealized as a combination of software and hardware.

The image acquisition module 201 acquires an image including an image of(the motion of) the user imaged by the above-mentioned camera 12B.

The operation detector 202 detects user operations (for example, motionsof a user's hand or finger, etc.) based on a user motion included in theimage acquired by the image acquisition module 201. The operationdetected by the operation detector 202 includes, for example, anoperation for displaying a user interface (which enables the user tooperate a 3D object) provided by, for example, the UI program, and anoperation on the user interface to deform a 3D object displayed on the3D display 12A.

If the operation detector 202 has detected an operation for displayingthe user interface, the UI generator 203 generates a user interfaceimage to be displayed on the 3D display 12A. The user interface (image)generated by the UI generator 203 has a rectangular frame shape and islocated around a 3D object currently displayed on the 3D display 12A. Itis sufficient if the rectangular frame surrounding the 3D objectincludes at least four linear lines (solid and/or broken lines)positioned above, below, leftward and rightward of the 3D object. Forinstance, part of the four corners of the frame may be curved or may notexist. Moreover, it is sufficient if the user interface has a shape thatcan surround the 3D object, namely, for example, a round shape.

The user interface generated by the UI generator 203 is displayed(drawn) on the 3D display 12A in a stereoscopic manner. This enables theuser to see, on the 3D display 12A, a 3D object and a rectangular userinterface positioned around the object.

If the operation detector 202 has detected an operation on the userinterface displayed (stereoscopically) on the 3D display 12A, thedeformation processor 204 deforms a 3D object surrounded by (therectangular frame of) the user interface in accordance with theoperation. Specifically, the deformation processor 204 rotates, enlargesor reduces the 3D object.

The 3D object deformed by the deformation processor 204 is displayed(redrawn) on the 3D display 12A. That is, the user can deform the 3Dobject, currently displayed on the 3D display 12A, by operating therectangular user interface.

Referring then to the flowchart of FIG. 4, a description will be givenof the procedure of processing performed by the electronic apparatus 10according to the embodiment.

Assume here that a plurality of 3D objects are displayed on the 3Ddisplay 12A, and the user operates the 3D objects.

In this case, the user can activate, for example, the camera 12B tothereby perform an operation (hereinafter, referred to as a firstoperation) of displaying a user interface on the 3D display 12A of theelectronic apparatus 10. The first operation includes, for example, auser (using the electronic apparatus 10) motion of holding up their handin front of the 3D display 12A (camera 12B).

Although in the embodiment, the user activates the camera 12B whenperforming the first operation, the camera 12B may always be activatedwhile the electronic apparatus 10 is in the ON state. If the camera 12Bis activated, the image acquisition module 201 continuously acquiresimages including motion of the user using the electronic apparatus 10(namely, motion of the user positioned before the electronic apparatus10). In the following description, images acquired by the imageacquisition module 201 will be referred to as captured images.

First, the operation detector 202 detects the first operation based on acaptured image (block B1). Specifically, the operation detector 202detects the first operation if the user's hand (the palm of the hand) isincluded in the captured image (that is, the motion of holding up a handis detected in the captured image).

After executing block B1, the UI generator 203 generates a rectangularuser interface (hereinafter, referred to as a macroframe UI) surroundinga plurality of 3D objects currently displayed on the 3D display 12A. Themacroframe UI generated by the UI generator 203 is spectroscopicallydisplayed (drawn) on the 3D display 12A so that it overlaps a pluralityof 3D objects in the same space (block B2). The macroframe UI may havethe same size as, for example, the 3D display 12A, or may be smallerthan the 3D display 12A. It is sufficient if the macroframe has a sizesufficient to enclose the plurality of 3D objects.

Next, the UI generator 203 generates a rectangular user interface(hereinafter, referred to as a microframe UI) surrounding one of theabove-mentioned 3D objects displayed on the 3D display 12A. Themicroframe UI generated by the UI generator 203 is spectroscopicallydisplayed (drawn) on the 3D display 12A so that it overlaps itssurrounding 3D object in the same space (block B3). It is sufficient ifthe microframe has a size that allows only one 3D object to besurrounded, and does not allow the other 3D objects to be surrounded.

In the embodiment, the position of a user's hand relative to a pluralityof 3D objects displayed on the 3D display 12A can be detected, based on,for example, the position and size of the hand of a user included in acaptured image (an image imaged by the camera 12B). The detectedposition of the user's hand includes the horizontal, vertical anddepth-wise positions (namely, the three-dimensional positions) of thecamera 12B. Thus, it is assumed that the 3D object surrounded by themicroframe UI displayed in block B3 is determined, based on the displaypositions of the 3D objects and the position of the user's hand in thefirst operation. Specifically, the UI generator 203 generates and drawsa microframe UI, which surrounds a 3D object displayed in a positionclosest to the position of the user's hand detected as described above,of a plurality of 3D objects currently displayed on the 3D display 12A.

Specific examples of display modes of the macroframe and microframe UIsdisplayed in blocks B2 and B3 will be described later.

If the macroframe and microframe UIs are displayed in blocks B2 and B3,the user can select a 3D object as an operation target by moving theirown hand.

In this case, the operation detector 202 detects the position of theuser's hand relative to the 3D object, based on the captured image asdescribed above (block B4).

The operation detector 202 selects (determines) a 3D object as anoperation target, based on the detected position of the user's hand(block B5).

Specifically, in block B5, if, for example, a user's hand has beendetected to be in a position for designating one of a plurality of 3Dobjects (position in which, for example, the hand overlaps the one 3Dobject) (more specifically, if such a motion of the hand as pointing theone 3D object has been detected), the operation detector 202 selects the3D object. In contrast, if, for example, a user's hand is not in aposition for designating one of the 3D objects (if, for example, theuser has made such a motion as pointing something other than the 3Dobjects), the operation detector 202 selects the plurality of 3D objects(namely, all 3D objects displayed on the 3D display 12A).

It is assumed that if a 3D object other than the 3D object surrounded bythe microframe UI displayed in block B3 is selected, the microframe UIis hidden, and another microframe UI that surrounds the selected 3Dobject is newly displayed. That is, the 3D object surrounded by themicroframe UI can be changed by moving a user's hand to a position fordesignating a desired 3D object.

In the description below, the 3D object selected in block B5 will bereferred to as a selected object, and the UI (the macroframe ormicroframe UI) that surrounds the selected object will be referred to aselected UI.

Next, the operation detector 202 determines whether the user has made anoperation (hereinafter, referred to as a second operation) of holdingthe selected UI or a similar operation, based on the captured image(block B6). In block B6, the second operation is determined to have beenperformed, if it is detected from the captured image that, for example,both the user's hands are positioned in positions corresponding torectangular frames that substantially oppose each other with a surfaceformed by the selected UI (hereinafter, referred to as a selected-UIformed surface) interposed therebetween.

The description below includes, for example, a sentence “the user holdsa UI (macroframe or microframe UI).” This sentence indicates a statewhere both the user's hands are (estimated to be) in positionssubstantially corresponding to positions on a UI (a rectangular frameUI) that oppose each other with the aforementioned spectroscopicallydisplayed UI-forming surface interposed therebetween.

If it is determined that the second operation has been performed (YES inblock B6), the UI generator 203 changes the color of the selected UI inorder to inform the user that the second operation has been performed(that is, the second operation on the selected UI has been detected)(block B7). Instead of this processing, in block B7, the shape of theselected UI may be changed, or a mark indicating that the secondoperation has been detected may be displayed. It is sufficient if thedetection of the second operation can be informed to the user.

In addition, in order to enhance the visibility of the selected UI andthe selected object, the UI (namely, the UI not selected in block B5)other than the selected one may be hidden after the second operation isdetected.

Next, the user can perform an operation (hereinafter, referred to as athird operation) on the selected UI for deforming the selected object.The third operation includes, for example, an operation of rotating theselected object (hereinafter, referred to as a rotational operation),and an operation of enlarging or reducing the selected object(hereinafter, referred to as an enlargement/reduction operation).Specific examples of the rotational operation and theenlargement/reduction operation will be described later.

In this case, the operation detector 202 detects (acquires) a change inthe position of the user's hand within a period in which the select UIis displayed, based on the captured image (block B9). The change in theposition of the user's hand is expressed by, for example,three-dimensional coordinate values.

Next, the operation detector 202 determines whether the third operationhas been performed based on changes in the detected positions of both ofthe user's hands (block B10).

If it is determined that no third operation is performed (NO in blockB10), the processing of block B12, which will be described later, isperformed.

In contrast, if it is determined that the third operation has beenperformed (YES in block B10), the deformation processor 204 deforms theselected object and the selected UI based on the third operation (i.e.,based on changes in both of the user's hands), and displays (draws) theresultant figures on the 3D display 12A (block B11).

If the third operation is a rotational operation, the selected objectand the selected UI are rotated about the center of rotation to bedisplayed, described later. At this time, the rotational angle of theselected object and the selected UI is computed (determined) based onchanges in the positions of both of the user's hands in the rotationaloperation.

In contrast, if the third operation is an enlargement/reductionoperation, the selected object and the selected UI are enlarged orreduced to be displayed. At this time, the enlargement or reductionratio of the selected object and the selected UI is computed(determined) based on changes in the positions of both the user's handsin the enlargement/reduction operation.

Next, the operation detector 202 determines whether an operation ofremoving the user's hands from the selected UI (hereinafter, referred toas a fourth operation) has been performed, based on the captured image(block B12). In block B12, if it is detected based on the captured imagethat, for example, both the user's hands are deviated (separated) frompositions substantially corresponding to positions on the selected UI(of a rectangular shape) that oppose each other with the selected UIinterposed therebetween, it is determined that the fourth operation hasbeen performed.

If it is determined that the fourth operation has been performed (YES inblock B12), the UI generator 203 cancels change of the color of the UI(returns, to the original color, the color of the UI changed) in blockB7, in order to inform the user that the fourth operation has beenperformed (that is, the fourth operation on the selected UI has beendetected) (block B13).

After executing block B13, the procedure is repeated from block B4.

In contrast, if it is determined in block B12 that the fourth operationhas not been performed (NO in block B12), the procedure is repeated fromblock B9. In this case, the user can continuously perform an operationfor deforming the selected object (namely, the third operation).

If it is determined in block B6 that the second operation has not beenperformed (NO in block B6), the user can perform an operation for hidingthe user interfaces (the macroframe and macroframe UIs) (hereinafter,referred to as a fifth operation). The fifth operation includes anoperation of lowering a user's hand (that is, an operation of moving theuser's hand to the outside of the imaging area of the camera 12B).

In this case, the operation detector 202 determines, based on thecaptured image, whether the fifth operation has been performed (blockB14).

If it is determined that the fifth operation has been performed (YES inblock B14), the macroframe and microframe UIs currently displayed on the3D display 12A are hidden (block B15). After executing block B15, theprocedure is finished.

In contrast, if it is determined that the fifth operation has not beenperformed (NO in block B14), the procedure is repeated from block B4.

In the example of FIG. 4, the macroframe and microframe UIs aredisplayed in blocks B2 and B3 after the first operation is detected inblock B1. However, this structure may be modified such that, forexample, after a 3D object is selected in block B5, a UI (macroframe ormicroframe UI) that surrounds the selected 3D object is displayed. Inthis modification, if, for example, block B4 and subsequent blocks arerepeated after block B13, the macroframe and microframe UIs may be oncehidden.

Moreover, if the first operation is performed with both hands (that is,the user holds up both hands), the macroframe UI may be displayed,using, as a default, a state where a plurality of 3D objects areselected. In contrast, if the first operation is performed by one handof the user, a microframe UI may be displayed, using, as a default, astate where a single 3D object is selected.

Further, in the processing of FIG. 4, the second operation is performedafter an operation target (a 3D object as the operation target) isselected in block B5. This structure may be modified such that theoperation target is selected by the second operation. In this case, ifthe user has performed an operation of holding the macroframe ormicroframe UI (second operation) in a state where the macroframe andmicroframe UIs are displayed, it is sufficient if block B7 andsubsequent blocks are executed using, as the operation target, a 3Dobject surrounded by a UI on which the operation has been performed.

Referring then to FIG. 5, a description will be given of the macroframeand microframe UIs displayed in blocks B2 and B3. In the example shownin FIG. 5, 3D objects 301 to 303 are supposed to be displayed as theplurality of 3D objects on the 3D display 12A.

In this case, as shown in FIG. 5, a macroframe UI 401 that includes(surrounds) all 3D objects 301 to 303 displayed on the 3D display 12A isdisplayed on the 3D display 12A. Further, a microframe UI 402 thatincludes (surrounds) 3D object 302 is displayed on the 3D display 12A.3D object 302 included by the microframe UI 402 is a 3D object closest,among 3D objects 301 to 303, to the position of a user's hand in thefirst operation.

In FIG. 5, although the macroframe and microframe UIs 401 and 402 arerectangular, they may have a shape (such as a circular shape) other thanthe rectangular shape.

In the embodiment, 3D objects 301 to 303 surrounded by the macroframe UI401 can be, for example, rotated in accordance with a user operation(rotational operation) on the macroframe UI 401, as described above. Inthis case, 3D objects 301 to 303 are rotated about the aforementionedcenter of rotation. To this end, in the macroframe UI 401 (namely, inthe rectangular frame of the user interface), axial lines indicating thecenter of rotation, which are used as reference lines to be referred tofor rotating 3D objects 301 to 303 surrounded by the frame UI 401, aredisplayed. The axial lines displayed in the macroframe UI 401 include ahorizontal axial line 401 a, a vertical axial line 401 b and adepth-wise axial line. In the embodiment, the intersection of theseaxial lines expresses the center of rotation. In FIG. 5, the depth-wiseaxial line is omitted for convenience.

By the display of the axial lines in the macroframe UI 401, the user caneasily grasp the center of rotation of 3D objects 301 to 303. Inaddition, it is assumed that the center of rotation is set, as adefault, at the center of the macroframe UI 401 (namely, at the centerof the space defined by the rectangular frame).

Although omitted in FIG. 5, the above-mentioned axial lines are assumedto be also displayed in each microframe UI 402.

The axial lines may be displayed when the macroframe and microframe UIsare displayed on the 3D display 12A in blocks B2 and B3, or may bedisplayed before the user performs a rotational operation (for example,after the processing of block B7).

Further, although in FIG. 5, it is assumed that only one microframe UIis displayed, a plurality of microframes 402 a to 402 c respectivelysurrounding 3D objects 301 to 303 may be displayed, for example, as isshown in FIG. 6. For instance, if 3D object 302 is selected in block B5shown in FIG. 4, microframe UI 402 b is made to be discriminated fromthe other UIs (i.e., macroframe UI 401 and microframe UIs 402 a and 402c) by changing the color of microframe UI 402 b surrounding 3D object302.

If only one microframe UI is displayed as shown in FIG. 5, the 3D objectsurrounded by this microframe UI can be changed (selected) by moving, asdescribed above, a user's hand to a position for designating a 3Dobject. Furthermore, in the embodiment, the depth-wise position (used asdepth data) of a user's hand can be detected based on an captured image,as described above.

Accordingly, even if, for example, two 3D objects 501 and 502 overlapeach other as shown in FIG. 7, it can be detected that the user hasmoved their hand to a position for designating 3D object 501 or 502, bycomparing the display positions of 3D objects 501 and 502 with thedepth-wise position of the user's hand.

As a result, in a case where microframe UI 402 surrounding 3D object 501is displayed, if the user has moved their hand to a position deeper(i.e., toward the 3D display 12A) than microframe UI 402 (or 3D object501), microframe UI 402 surrounding 3D object 502 can be displayed. Inother words, the 3D object surrounded by microframe UI 402 can bechanged from 3D object 501 to 3D object 502.

Further, if another 3D object 501 overlaps 3D object 502, surrounded bymicroframe UI 402, in front of 3D object 502 as shown in FIG. 7, it maybe displayed in a see-through way in order to raise the visibility of 3Dobject 502.

Referring then to FIG. 8, the rotational operation included in the thirdoperation will be described in detail. Assume here that the macroframeUI 401 is a selected UI.

As is described referring to FIG. 4, the user performs the secondoperation (in this case, such an operation as holding ends of themacroframe UI 401) before the rotational operation (third operation).Accordingly, if the rotational operation is performed, it is assumedthat both of the user's hands are in positions 601 and 602 correspondingto positions on (the rectangular shape of) the macroframe UI 401 thatsubstantially oppose each other with the forming surface of themacroframe UI 401 interposed therebetween, as is shown in the upperportion of FIG. 8.

In this state, it is assumed that the operation detector detects therotational operation when detecting that one of the user's hands (forexample, the left hand) has been moved in a direction not parallel tothe forming surface of the macroframe UI 401 (for example, the depthdirection), and the other hand (for example, the right hand) of theuser's hands has been moved in a direction (for example, toward thefront) opposite to the first-mentioned direction (more specifically,when detecting changes in the positions of the user's handscorresponding to the above-mentioned movements of the left and righthands). Conceptually, if such an operation as pushing one of the user'shands holding (the rectangular shape of) the macroframe UI 401, andpulling the other hand is detected, the rotational operation isdetected.

If such a rotational operation is detected, 3D objects (selectedobjects) 301 to 303 and the macroframe UI 401 (namely, the entire spacedefined by the macroframe UI 401) can be rotated about, for example,axial line 401 a as a rotation axis at an angle corresponding to thechanges in the positions of the user's hands caused by the rotationaloperation to be displayed, as is shown in the lower portion of FIG. 8.

Moreover, in the example of FIG. 8, the user holds, using both hands,positions 601 and 602 on the macroframe UI 401. However, the positionsin which the user holds the macroframe UI 401 are not limited topositions on the left and right sides of the macroframe UI 401. They maybe positions on the upper and lower sides of the macroframe UI 401. Thatis, if, for example, such an operation (rotational operation) asdepth-wise pushing one of the user's hands holding the upper and lowersides of the macroframe 401, and pulling the other hand is detected, 3Dobjects 301 to 303 and the macroframe UI 401 can be rotated about, forexample, axial line 401 b as a rotation axis to be displayed.

The rotation of the 3D objects is not limited to those about axial lines401 a and 401 b as the rotation axes, but can be three-dimensionallyrotated in an arbitrary direction with respect to the centers ofrotation indicated by the axial lines.

It is sufficient if the hands of the user are in positions where changesin the positions of the hands of the user can be detected for detectingthe above-mentioned rotational operation. For instance, the positionsare not limited to those opposing each other with the forming surface ofthe macroframe UI 401 interposed therebetween. More specifically, if,for example, the macroframe UI 401 is of a rectangular shape, thepositions of the user's hands may be on adjacent sides or a single sideof (the rectangular shape of) the macroframe UI 401.

Although FIG. 8 is directed to the case where the selected UI is themacroframe UI 401, the rotational operation can be also performed if theselected UI is the microframe UI 402.

Referring now to FIG. 9, a detailed description will be given of theenlargement/reduction operation included in the above-described thirdoperation. Assume here that the microframe UI 402 surrounding 3D object302 is the selected UI. The enlargement/reduction operation includes anenlargement operation and a reduction operation.

As in the rotational operation described referring to FIG. 8, the secondoperation (in this case, an operation of holding the microframe IU 402)is performed by the user, before the enlargement/reduction operation(third operation) is performed by the user. Accordingly, if theenlargement/reduction operation is performed, it is assumed that thehands of the user are in positions 701 and 702 corresponding topositions on (the rectangular shape of) the microframe UI 402 thatsubstantially oppose each other with the forming surface of themicroframe UI 402, as is shown in the upper portion of FIG. 9.

In this case, it is assumed that the operation detector 202 detects theenlargement operation when detecting that one (for example, the lefthand) of the user's hands has been moved in a direction substantiallyparallel to the forming surface of the microframe UI 402 (for example,in the left direction), and the other hand (for example, the right hand)of the user's hands has been moved in a direction (for example, in theright direction) opposite to the first-mentioned direction (morespecifically, when detecting changes in the positions of the user'shands corresponding to the above-mentioned movements of the left andright hands). Conceptually, if such an operation as horizontallystretching (the rectangular shape of) the microframe UI 402 (that is,enlarging the microframe UI 402) is detected, the enlargement operationis detected.

If such an enlargement operation is detected, 3D object 302 and themicroframe UI 402 can be enlarged with a ratio corresponding to changesin the positions of the user's hands caused by the enlargement operationto be displayed, as is shown in the lower portion of FIG. 9.

FIG. 9 shows the enlargement operation. Similarly, if it is detectedthat the user's left hand has been moved rightward and the user's righthand has been moved leftward, the operation detector 202 detects thereducing operation. In other words, if a user operation of horizontallyreducing the microframe UI 402 is detected, the reducing operation isdetected.

If such a reducing operation is detected, 3D object 302 and themicroframe UI 402 can be reduced with a ratio corresponding to changesin the positions of the user's hands caused by the reducing operation tobe displayed.

Although FIG. 9 is directed to the case where the user holds positions701 and 702 on the microframe UI 402, using both hands, the positions inwhich the user holds the microframe are not limited to positions on theleft and right sides of the microframe UI 402. They may be positions onthe upper and lower sides of the same. That is, even if, for example,such a user operation (enlargement/reduction operation) of verticallyenlarging (or vertically reducing) the microframe UI 402 has beendetected, with the upper and lower sides of the microframe UI 402 heldby the user, 3D object 302 and the microframe UI 402 can be enlarged (orreduced) to be displayed.

It is sufficient if the user's hands are in positions where changes inthe user's hands can be detected for detecting the above-describedenlargement/reduction operation. For instance, the positions of theuser's hands are not limited to the positions that oppose each otherwith the forming surface of the microframe 402 interposed therebetween.

Although FIG. 9 is directed to the case where the selected UI is themicroframe UI 401, the enlargement/reduction operation can be alsoperformed if the selected UI is the macroframe UI 401. In other words,3D objects 301 to 303 and the macroframe UI 401 (namely, the entirespace defined by the rectangular shape of the macroframe UI 401) can beenlarged or reduced by the enlargement/reduction operation.

Further, in the embodiment, the rotational operation and theenlargement/reduction operation are detected if the user holds aselected UI with both hands. However, the embodiment may be modifiedsuch that if the user performs such an operation as holding (anarbitrary position on) the selected UI, with one hand, a differentoperation is detected.

More specifically, it is assumed that the user holds, using their onehand, position 801 on the right side of the microframe UI 402 (selectedUI), as is shown in, for example, the upper portion of FIG. 10. It isalso assumed that the microframe UI 402 surrounds 3D object 302.

In this case, when detecting that, for example, the user's right handhas been moved substantially parallel to the forming surface of themicroframe UI 402 (for example, horizontally or vertically) (morespecifically, when detecting a change in the position of the user'sright hand corresponding to the movement of the right hand), theoperation detector 202 is assumed to detect an operation of moving 3Dobject 302 and the microframe UI 402 (hereinafter, referred to as amoving operation).

If such a moving operation has been detected, 3D object 302 and themicroframe UI 402 can be moved in accordance with the change in theposition of the user's right hand due to the moving operation to bedisplayed.

As described above, the embodiment is directed to the case where theuser horizontally or vertically moves the microframe UI 402, using theirone hand. However, if, for example, such an operation as moving themicroframe UI 402 backward or forward (closer to or away from the user)is performed, 3D object 302 and the microframe UI 402 may be rotatedabout the aforementioned center of rotation to be displayed.

Moreover, although FIG. 10 is directed to the case where the selected UIis the microframe UI 402, the same moving operation as the above can beperformed if the selected UI is the macroframe UI 401. In other words,the moving operation can move 3D objects 301 to 303 and the macroframeUI 401 (namely, the entire space defined by the rectangular shape of themacroframe UI 401).

In the above-described embodiment, if a 3D object (first 3D object) hasbeen selected from a plurality of 3D objects displayed on the 3D display12A, a rectangular microframe UI (first user interface) surrounding this3D object is stereoscopically displayed on the 3D display, and the 3Dobject and the microframe UI are rotated through an angle (first angle)corresponding to changes in the positions of the user's hands that occurwhen the microframe UI is displayed. Further, if a plurality of 3Dobjects (first and second 3D objects) have been selected, therectangular macroframe UI (second user interface) surrounding the 3Dobjects is stereoscopically displayed on the 3D display, and the 3Dobjects and the macroframe UI are rotated through an angle (secondangle) corresponding to changes in the positions of the user's handsthat occur when the macroframe UI is displayed.

More specifically, in the embodiment, after it is detected based on acaptured image that the user's hands are in positions corresponding topositions on a rectangular microframe UI that oppose each other with theforming surface (first surface) of the microframe UI, if it is detectedbased on the captured image that one (first hand) of the user's hands ismoved in a direction (first direction) not parallel to the formingsurface of the microframe UI, and that the other hand (second hand) ismoved in a direction (second direction) opposite to the first direction,a 3D object surrounded by the microframe UI and the microframe UI arerotated. Moreover, after it is detected based on a captured image thatthe user's hands are in positions corresponding to positions on arectangular macroframe UI that oppose each other with the formingsurface (second surface) of the macroframe UI, if it is detected basedon the captured image that one of the user's hands is moved in adirection (third direction) not parallel to the forming surface of themacroframe UI, and that the other hand is moved in a direction (fourthdirection) opposite to the third direction, 3D objects surrounded by themacroframe UI and the macroframe UI are rotated.

In the embodiment, the above-described structure enables the user torotate a desired 3D object by moving both hands in association with themicroframe or macroframe UI. Thus, the embodiment provides a userinterface capable of intuitively, intelligibly and easily rotating 3Dobjects. Further, since an operation of rotating a 3D object isperformed on a rectangular UI (a microframe UI or the macroframe UI)located around the 3D object, the 3D object as an operation target isnot hidden by the hands of the user even during the operation, whichenhances the visibility and operability of the operation target.

Furthermore, in the embodiment, a 3D object surrounded by a microframeUI is selected when it is detected, based on a captured image, that auser's hand is in a position (first position) for designating the 3Dobject. Moreover, a plurality of 3D objects surrounded by the macroframeUI are selected when it is detected based on a captured image that auser's hand is in a position (second position) for designating the 3Dobjects. In the embodiment, the above-described structure enables anoperation target to be easily changed (switched) among a plurality of 3Dobjects displayed on the 3D display 12A.

Yet further, in the embodiment, after it is detected based on a capturedimage that the user's hands are in positions corresponding to positionson a rectangular microframe UI that oppose each other with the formingsurface of the microframe UI, if it is detected based on the capturedimage that one of the user's hands is moved in a direction (firstdirection) substantially parallel to the forming surface of themicroframe UI, and that the other hand is moved in a direction (seconddirection) opposite to the first direction, a 3D object surrounded bythe microframe UI and the microframe UI are enlarged or reduced, basedon changes in the positions of the user's hands. Also, after it isdetected based on a captured image that the user's hands are inpositions corresponding to positions on a rectangular macroframe UI thatoppose each other with the forming surface of the macroframe UI, if itis detected based on the captured image that one of the user's hands ismoved in a direction (third direction) substantially parallel to theforming surface of the macroframe UI, and that the other hand is movedin a direction (fourth direction) opposite to the third direction, a 3Dobject surrounded by the macroframe UI and the macroframe UI areenlarged or reduced, based on changes in the positions of the user'shands.

In the embodiment, the above-described structure enables a desired 3Dobject to be enlarged or reduced, as well as rotated, by causing theuser to move both hands with respect to the microframe or macroframe UI.

Further, in the embodiment, axial lines (first axial lines) indicating acentral point (center of rotation), about which a 3D object surroundedby a microframe UI and the microframe UI are rotated, are displayed in(the rectangular shape of) the microframe UI. Furthermore, axial lines(second axial lines) indicating a central point (center of rotation),about which a 3D object surrounded by a macroframe UI and the macroframeUI are rotated, are displayed in (the rectangular shape of) themacroframe UI. In the embodiment, this structure enables the center ofrotation of a 3D object to be presented for the user to thereby supporta user operation (3D operation).

In the embodiment, a 3D object or 3D objects surrounded by a microframeor macroframe UI are rotated about the center of rotation set as adefault at the center of (the space defined by the rectangular shape of)the microframe or macroframe UI, as described above. The embodiment canbe modified such that the center of rotation can be changed inaccordance with a user operation.

Referring then to FIG. 11, a user operation for changing the center ofrotation will be described in detail. Assume that the 3D display 12Adisplays a plurality of 3D objects 301 to 303 and the macroframe UI 401as shown in the upper portion of FIG. 11. Assume also that themacroframe UI 401 displays vertical axial line 401 a, horizontal axialline 401 b, and depth-wise axial line 401 c. In this case, theintersection of axial lines 401 a to 401 c represents the center ofrotation.

When changing the center of rotation, the user firstly sets, forexample, the electronic apparatus 10 in a mode for changing the centerof rotation (hereinafter, referred to as a change mode), and thenperforms such an operation (hereinafter, referred to as a changeoperation) as designating the position of the center of rotation, usingone of the hands (for example, an operation of pointing the position ofthe center of rotation).

Assuming that position 901 shown in the lower portion of FIG. 11 isdesignated by the change operation, position 901 is set as the center ofrotation. Position (three-dimensional position) 901 designated by thechange operation can be determined by the operation detector 202 basedon a captured image, as described above. In this case, axial lines 401 ato 401 c are redrawn to intersect each other at (the position set as)the center of rotation 901.

If, for example, the above-mentioned rotation operation has beenperformed on the macroframe UI 401, 3D objects 301 to 303 and themacroframe UI 401 are rotated about the center of rotation 901.

The above-described structure capable of changing the position of thecenter of rotation enables the user to set the center of rotation in adesired position by an intuitive operation, which further enhances theoperability.

As described above, in the change operation shown in FIG. 11, theposition (three-dimensional position) designated by a user's hand can beset as the center of rotation. However, the designated position isdetermined based on a captured image. If a three-dimensional position isdetermined based on a captured image, an error may easily occur in thedepth-wise direction (front-back direction), compared to the horizontaland vertical directions. Therefore, in the above-described changeoperation, the user may be unable to change the center of rotation to adesired position.

In view of the above, the embodiment has a structure capable ofadjusting the depth-wise position of the center of rotation, usinganother operation. More specifically, it is assumed that in a statewhere the user holds positions 902 and 903 on the macroframe UI 401shown in FIG. 12, using both hands, if a user operation of, for example,moving both hands toward the user has been detected, the center ofrotation can be moved toward the user as shown in FIG. 13. FIG. 13 is aconceptual diagram (an overhead view) obtained when a plurality of 3Dobjects 301 to 303 and the macroframe UI 401 (namely, the macrospacedefined by the macroframe UI 401) are viewed from above.

Although not illustrated, the center of rotation can also be moveddepth-wise if, for example, such a user operation as moving both handstoward the display.

In the above-described structure, if, for example, the position of thecenter of rotation changed by the change operation is deviated, it canbe adjusted to a desired position for the user.

If the position of the center of rotation is changed as described above,a numerical value, for example, indicating the changed position may bedisplayed to enable the user to grasp the changed position.

In the embodiment, a desired 3D object can be easily rotated, moved,enlarged and reduced and the center of rotation can be easily changed,by an intuitive operation, such as pulling, pushing, and positiondesignation (pointing), as is described above.

It is described above that the center of rotation is changed when thechange mode is set in the electronic apparatus 10. However, if theoperation shown in FIG. 12 is performed where the change mode is notset, the format of display of a 3D object, such as the degree ofspectroscopic display (i.e., the degree of projection), may be changed.

In the embodiment, if a plurality of cameras (for example, two-pointcameras) are used as the camera 12B, the depth-wise positions of theuser's hands can be recognized more accurately. In this case, operationsintended by the user can be more accurately detected, whereby theoperability of the apparatus is enhanced. In order to enhance theaccuracy of recognition of the depth-wise position of each hand, a depthsensor 12C may be employed as well as the camera 12B, as is shown inFIG. 14.

Although the embodiment is directed to the case where the electronicapparatus 10 is realized as a notebook computer, the electronicapparatus 10 may be another type of electronic apparatus, such as atelevision or a computer monitor. It is sufficient if the electronicapparatus 10 can display 3D objects.

Each of the functions described in the embodiment may be realized by acircuit (processing circuit). The processing circuit includes, forexample, a programmed hardware processor, such as a central processingunit (CPU). The processor executes a computer program (a group ofcommands) stored in a memory, to thereby realize the describedfunctions. The processor may be a microprocessor including electricalcircuits. The processing circuits include a digital signal processor(DSP), an application-specific integrated circuit (ASIC), amicrocontroller, a controller, and other electronic circuit components.Each of the components other than the CPU described in the embodimentmay also be realized by the processing circuits.

Moreover, since various types of processing in the embodiment can berealized by computer programs, the same advantages as those of theembodiment can be easily obtained simply by installing the computerprograms to a computer through a computer-readable storage mediumstoring them and executing them.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An electronic apparatus comprising: a displayconfigured to display a three-dimensional object; a camera configured tocapture an image including hands of a user; and a hardware processorconfigured to: display at least a first three-dimensional object and asecond three-dimensional object on the display; spectroscopicallydisplay a first user interface of a rectangular shape surrounding thefirst three-dimensional object and the second three-dimensional objecton the display, when a first operation is detected based on the capturedimage obtained from the camera and including a movement of the hands ofthe user; upon detecting, based on the captured image obtained from thecamera and including a position and a size of the hands of the user,that one of the first three-dimensional object and the secondthree-dimensional object is selected, spectroscopically display a seconduser interface of a rectangular shape surrounding the selectedthree-dimensional object; generate a display to allow the user torecognize that one of the first user interface and the second userinterface is a selected user interface (UI), when a second operation isdetected based on the captured image obtained from the camera andincluding the hands of the user; and when a third operation for theselected UI is detected based on the captured image obtained from thecamera and including changes in positions of the hands of the user,calculate a rotational angle, an enlargement ratio or a reduction ratioof a selected object included in the selected UI based on the changes,thereby deforming the selected object.
 2. The electronic apparatus ofclaim 1, wherein the hardware processor is configured to rotate theselected object corresponding to the first user interface and the firstuser interface at an angle corresponding to the changes, when it isdetected, based on the captured image, that the user's hands arepositioned in the first user interface, and then it is detected, basedon the captured image, that a first hand of the user is moved in a firstdirection not parallel to the first surface and a second hand of theuser is moved in a second direction opposite to the first direction; andthe hardware processor is configured to rotate the selected objectcorresponding to the second user interface and the second user interfaceat the angle corresponding to the changes, when it is detected, based onthe captured image image, that the user's hands are positioned in thesecond user interface, and then it is detected, based on the capturedimage, that the first hand of the user is moved in a third direction notparallel to the second surface and the second hand of the user is movedin a fourth direction opposite to the third direction.
 3. The electronicapparatus of claim 1, wherein the hardware processor is configured toenlarge or reduce the selected object corresponding to the first userinterface and the first user interface with a ratio corresponding to thechanges, when it is detected, based on the captured image, that theuser's hands are positioned in the first user interface, and then it isdetected, based on the captured image, that a first hand of the user ismoved in a first direction relative to the first surface and a secondhand of the user is moved in a second direction opposite to the firstdirection; and the hardware processor is configured to enlarge or reducethe selected object corresponding to the second user interface and thesecond user interface with the ratio corresponding to the changes, whenit is detected, based on the captured image, that the user's hands arepositioned in the second user interface, and then it is detected, basedon the captured image, that the first hand of the user is moved in athird direction relative to the second surface and the second hand ofthe user is moved in a fourth direction opposite to the third direction.4. The electronic apparatus of claim 2, wherein the hardware processoris configured to display, in the rectangular shape of the first userinterface, first axial lines indicating a center of rotation about whichthe selected object corresponding to the first user interface and thefirst user interface are rotated; and the hardware processor isconfigured to display, in the rectangular shape of the second userinterface, second axial lines indicating a center of rotation aboutwhich the selected object corresponding to the second user interface andthe second user interface are rotated.
 5. The electronic apparatus ofclaim 1, wherein if the user's hands are detected in the captured image,the hardware processor is configured to assume that the first and secondthree-dimensional objects are selected, and display the first userinterface.
 6. The electronic apparatus of claim 4, wherein the hardwareprocessor is configured to change a position of the center of rotation,based on a position in the rectangular shape of the first user interfacedesignated by a hand of the user included in the captured image.
 7. Theelectronic apparatus of claim 1, wherein the hardware processorcomprises: a module configured to display the first and secondthree-dimensional objects on the display; a module configured tostereoscopically display, on the display, the first user interface ofthe rectangular shape surrounding the first three-dimensional object andthe second three-dimensional object, when the first operation isdetected based on the captured image obtained from the camera andincluding the movement of the hands of the user; a module configured to,upon detecting, based on the captured image obtained from the camera andincluding the position and the size of the hands of the user, that oneof the first three-dimensional object and the second three-dimensionalobject is selected, stereoscopically display, on the display, the seconduser interface of the rectangular shape surrounding the selectedthree-dimensional object; a module configured to perform display toallow the user to recognize that one of the first user interface and thesecond user interface is the selected UI, when the second operation isdetected based on the captured image obtained from the camera andincluding the hands of the user; and a module configured to, when thethird operation for the selected UI is detected based on the capturedimage obtained from the camera and including the changes in thepositions of the hands of the user, calculate the rotational angle, theenlargement ratio or the reduction ratio of the selected object includedin the selected UI based on the changes, thereby deforming the selectedobject.
 8. A method comprising: displaying at least a firstthree-dimensional object and a second three-dimensional object on adisplay; spectroscopically displaying a first user interface of arectangular shape surrounding the first three-dimensional object and thesecond three-dimensional object on the display, when a first operationis detected based on a captured image obtained from a camera andincluding a movement of hands of a user; upon detecting, based on thecaptured image obtained from the camera and including a position and asize of the hands of the user, that one of the first three dimensionalobject and the second three-dimensional object is selected,spectroscopically displaying a second user interface of a rectangularshape surrounding the selected three-dimensional object on the display;performing display to allow the user to recognize that one of the firstuser interface and the second user interface is a selected userinterface (UI), when a second operation is detected based on thecaptured image obtained from the camera and including the hands of theuser; and when a third operation for the selected UI is detected basedon the captured image obtained from the camera and including changes inpositions of the hands of the user, calculating a rotational angle, anenlargement ratio or a reduction ratio of a selected object included inthe selected UI based on the changes, thereby deforming the selectedobject.
 9. The method of claim 8, further comprising: rotating theselected object corresponding to the first user interface and the firstuser interface at an angle corresponding to the changes, when it isdetected, based on the captured image, that the user's hands arepositioned in the first user interface, and then it is detected, basedon the captured image, that a first hand of the user is moved in a firstdirection not parallel to the first surface and a second hand of theuser is moved in a second direction opposite to the first direction; androtating the selected three-dimensional object corresponding to thesecond user interface and the second user interface at the anglecorresponding to the changes, when it is detected, based on the capturedimage, that the user's hands are positioned in the second userinterface, and then it is detected, based on the captured image, thatthe first hand of the user is moved in a third direction not parallel tothe second surface and the second hand of the user is moved in a fourthdirection opposite to the third direction.
 10. The method of claim 8,further comprising: enlarging or reducing the selected objectcorresponding to the first user interface and the first user interfacewith a ratio corresponding to the changes upon detecting, based on thecaptured image, that the user's hands are positioned in the first userinterface, and then it is detected, based on the captured image, that afirst hand of the user is moved in a first direction relative to thefirst surface and a second hand of the user is moved in a seconddirection opposite to the first direction; and enlarging or reducing theselected object corresponding to the second user interface and thesecond user interface with the ratio corresponding to the changes, whenit is detected, based on the captured image, that the user's hands arepositioned in the second interface, and then it is detected, based onthe captured image, that the first hand of the user is moved in a thirddirection relative to the second surface and the second hand of the useris moved in a fourth direction opposite to the third direction.
 11. Themethod of claim 9, further comprising: displaying, in the rectangularshape of the first user interface, first axial lines indicating a centerof rotation about which the selected object corresponding to the firstuser interface and the first user interface are rotated; and displaying,in the rectangular shape of the second user interface, second axiallines indicating a center of rotation about which the selectedthree-dimensional object corresponding to the second user interface andthe second user interface are rotated.
 12. The method of claim 8,wherein if the user's hands are detected in the captured image, it isassumed that the first and second three-dimensional objects areselected, and the first user interface is displayed.
 13. The method ofclaim 11, wherein a position of the center of rotation is changed basedon a position in the rectangular shape of the first user interfacedesignated by a hand of the user included in the captured image.